1. Field of the Invention
The present invention is a method and system to provide an automatic measurement of people's responses to dynamic digital media, based on changes in their facial expressions and attention to specific content.
2. Background of the Invention
Digital media is becoming more widespread and increasingly important in the realm of the commercial advertisement business. Displaying information about products and services through digital media is effective due to it's capability to attract consumers and deliver information through rich audiovisual contents. Digital media also has flexibility in producing the content and ease of delivering the content through automatic programming on a computer network.
As the number of screens displaying digital media increases, the task of assessing the effectiveness of the content toward customer conversion and increased sales revenue is becoming very challenging. It is mainly due to the sheer number of the displays and also to the complexity of the programming content to make individual assessments. Moreover, the true effectiveness of media content lies beneath the human mind—in what way consumers are attracted to the display, whether they react positively or negatively to the advertised material, and how the mental changes affect their perspective about the products and the purchase decisions.
The present invention first provides a method to measure the response of the digital media audience to a given media content. While measuring the changes in the human mind is a very difficult task, apparent changes in people's attention and facial expressions reflect the dynamics of their mental states. We designed an automatic method to estimate the changes in mental state through these visual cues. Video images capturing the media audience will be the input to the system, and the system will employ computer vision algorithms to process the facial images of the people in the audience to recognize their changes in attention and facial expressions.
On the other hand, the complexity of digital media content also poses challenges to the task of media measurement. Digital media is not static advertisements, such as product banners or signs; it continuously changes its content—visual scenes, texts, or animations within its programming. The changes in content are designed to deliver information about products in an effective manner. Therefore, measuring the overall response to the media programming would be only a subset of the task; measuring the effectiveness of the sub-content that appears within the spatial and temporal extent of the program will provide further insight into the media measurement, so that the digital media production can focus on individual elements to improve the overall effectiveness.
To meet this goal, the present invention measures the changes in the mental states of the media audience for individual content presented both in spatial and temporal extent of the content material. More specifically, it estimates at which screen position each person in the audience is watching, and recognizes the emotional changes of the person toward that specific sub-content. For example, a person may stay neutral through the video content of the programming until a certain scene appears that appeals to him in a positive way. In another scenario, a person may like the overall product descriptions but can become negative when other information is revealed in the form of text at a certain portion of the screen.
Recent developments in computer vision and artificial intelligence technology make it possible to recognize the human response to digital media display based on images. Facial image analysis has especially matured, so that faces can be detected and tracked from video images, and the motion of the facial features can also be estimated. The facial appearance changes due to facial expressions can be measured to estimate the internal emotional state of a person. The proposed invention aims to solve the problem under realistic scenarios, where people show their natural reaction toward digital media display. While it is not entirely possible to estimate the mental state of a person based solely on apparent changes, the changes in the person's facial expression often carry highly relevant information about the person's response. The facial expression is recognized to be translated into six emotion categories: Happiness, Sadness, Surprise, Anger, Disgust, and Fear. These categories are known to reflect the more fundamental affective states of the human mind: Arousal, Valence, and Stance. This invention assumes that these affective states, if estimated, provide information more directly relevant (than do the six emotion categories) to the recognition of people's responses toward certain visual elements. For example, the degree of valence directly reveals the positive or negative attitude. The changes in emotional state will then render a trajectory in the three-dimensional affect space. Under the context of temporally changing media content, the emotion trajectory represents the changes in the mental state of a person in relation to the displayed content.
It is also possible to measure the information as to at which screen position a person is watching—the gaze of the person. The measurement of gaze is one way to measure the person's interest and attention; the person's gaze reveals which material showing on the screen is capturing his/her attention at the moment. The gaze target—the location within the screen where the person's gaze is fixated—is estimated by measuring the eye gaze of the person as well as the facial pose of the person; the eye gaze of a person is defined as the orientation of the person's gaze relative to the person's face. An automated analysis of the person's image captured from at least one camera provides the measurement for estimating the gaze target. The cameras are placed and oriented so that they can capture the faces of potential viewers of the media display; the cameras are typically placed near the display. The estimated positions of the irises relative to the eyes, along with the estimated head orientation, reveal the shopper's gaze direction. However, because of the nonlinear way that different facial poses affect the appearance changes in the eye image due to the eye gaze, a machine learning-based method is introduced to find the facial pose-dependent gaze direction estimation. The final gaze target is estimated based on the estimated gaze direction and the person position (more specifically, the head position). The media screen is divided into target grids, so that the gaze target estimation can identify at which target grid a person is currently watching. The resolution of the gaze target estimation is determined by the number of target grids on the screen; the target grids are determined from the accuracy of the gaze target estimation.
Once the emotional state and the gaze target are determined, these two kinds of measurements are used to associate the emotional response to a specific content that the person is consuming. It is another crucial feature of the present invention to derive such information from these two modalities. The changes in emotional state are estimated in the form of temporal emotion trajectories in affect space. The estimate of visual attention will have the form of temporal changes of the gaze target in terms of (X, Y) coordinates on the screen. The combination—emotional changes localized to screen positions—will be called the “spatiotemporal emotional response map.” At each point (X, Y, t) in the spatiotemporal domain, the collections of three-dimensional coordinates (in the affect space) of emotional responses of people watching the content displayed at (X, Y) will be recorded. The spatial resolution of the target grid on the screen can range from a resolution as rough as “upper left to upper right” or “lower left to lower right,” up to a much higher resolution, depending on the accuracy of the gaze target estimation algorithm under the given environment. The collected spatiotemporal emotional response map can be further analyzed to derive useful rating data corresponding to the specific media content.
There have been prior attempts for automatically recognizing the facial expression of a person from video images.
In U.S. Pat. No. 5,774,591 of Black, et al. (hereinafter Black), the motions of the facial features due to expression are estimated by computing an explicit parametric model of optical flow. The facial feature motions are translated into mid-level predicates, which in turn are used to determine the expression categories. The proposed invention utilizes emotion-sensitive features that extract feature shape changes implicitly, just to be fed to a learning machine to estimate the facial muscle action. In U.S. Pat. No. 6,072,496 of Guenter, et al. (hereinafter Guenter), the facial actions are estimated in terms of very involved three-dimensional mesh model by tracking a set of dedicated marker points. The present invention strives to estimate the shape change of the facial features just enough to determine the facial muscle actions, without using any artificial markers. U.S. Pat. No. 6,879,709 of Tian, et al. (hereinafter Tian-1) only aims to detect emotionless faces, while the present invention tries to estimate the change of expressions in a space representing the whole range of human emotions. In U.S. Pat. Appl. Pub. No. 2007/0265507 of de Lemos (hereinafter de Lemos), mostly eye tracking estimates are used to assess the degree of attention and the target of attention within the visual stimulus. The present invention shares a similar goal of estimating human response in relation to a given visual target, but introduces a different focus on the measurement of whole facial feature shapes to determine the emotional changes, with specific technical methods to estimate the facial actions, emotional changes, and finally the response. The response to the changing media content is also analyzed based on the emotion trajectory in relation to the time and screen positions of the specific media sub-content. “Measuring facial expressions by computer image analysis,” Psychophysiology, Vol. 36, Issue 2, by Bartlett, et al. (hereinafter Bartlett) aims to estimate upper facial Action Units utilizing the holistic, feature-based, and motion (flow)-based image representation and a neural network-based learning of the representation. “Recognizing Action Units for Facial Expression Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 2, by Tian, et al. (hereinafter Tian-2) also estimates parametric models of facial feature shapes, and employs neural networks to learn the mapping to the facial Action Units. The present invention also estimates the facial Action Units in an exemplary embodiment of facial muscle actions, and utilizes a learning machine to find a mapping from the image representation to the muscle actions. However, the present invention utilizes an emotion-sensitive feature extraction scheme, which is different from Bartlett or Tian-2. The present invention also utilizes a novel scheme to localize a face and its facial features, while in Bartlett the faces are assumed to be aligned. In “Active and dynamic information fusion for facial expression understanding from image sequences,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, Issue 5, by Zhang, et al. (hereinafter Zhang), the dynamic change of facial expressions is recognized by a series of methods starting from IR-based eye detection, and facial feature detection based on the eye detection. The facial Action Units recognition is based on deterministic correspondence. The present invention employs a novel combination of the face detection, localization, and facial feature localization. The mapping from the facial features shapes to the facial muscle actions is learned by training on a large number of samples.
There have been prior attempts for automatically measuring the audience response to displayed objects or media.
U.S. Pat. No. 7,113,916 of Hill (hereinafter Hill) aims to measure consumer reaction to marketing stimulus, whose goal is shared by the present invention. However, Hill lists interviewing and manual video coding as tools for collecting opinions and facial expressions. U.S. Pat. No. 7,120,880 of Dryer, et al. (hereinafter Dryer) proposes a system utilizing a host of measurement modalities, such as facial expression, head gesture, or speech, to assess the level of interest to media content. U.S. Pat. No. 7,233,684 of Fedorovskaya, et al. (hereinafter Fedorovskaya-1) and U.S. Pat. Appl. Pub. No. 2003/0156304 of Fedorovskaya, et al. (hereinafter Fedorovskaya-2) propose systems measuring affective information based on the visual image or physiological signal of a person and associating the affective information with the image and person, respectively. Hill, Dryer, Fedorovskaya-1, and Fedorovskaya-2 all propose overall systems, without introducing a very specific novel technical means to achieve the recognition of the response or affective information. The present invention introduces novel technology to automatically extract relevant information from the raw image data and recognize the emotional changes as well as the target of attention. U.S. Pat. Appl. Pub. No. 2003/0032890 of Hazlett, et al. (hereinafter Hazlett) proposes an emotional response measurement based on electromyographic signal of facial muscles, while the present invention processes common visual signal to make the same kind of measurement. U.S. Pat. Appl. Pub. No. 2007/0094172 of Thaler (hereinafter Thaler) presents a rating system utilizing neural networks, without specific reference to how the input data to the neural network is generated. The present invention also uses learning machines such as neural networks, but the learning machines are trained to process feature vectors that are extracted from video images following novel and specific procedures.
There have been prior attempts for automatically estimating the gaze direction of a human observer.
In U.S. Pat. No. 5,797,046 of Nagano, et al. (hereinafter Nagano), the gaze direction is estimated based on the optical signal of the light reflected by the iris, and on the stored personal signature of the reflection. In U.S. Pat. No. 5,818,954 of Tomono, et al. (hereinafter Tomono), the measured position of the iris relative to the measured facial coordinate is used to estimate the gaze. In U.S. Pat. No. 6,154,559 of Beardsley (hereinafter Beardsley), the gaze target is recognized based on the measurement of the head pose and the correlation between a known visual target and the head pose, using the head pose histogram of frequent gaze targets. In U.S. Pat. No. 6,246,779 of Fukui, et al. (hereinafter Fukui), the gaze is estimated by comparing the measured facial image feature pattern against the stored facial image feature patterns, using neural networks. In U.S. Pat. No. 7,043,056 of Edwards, et al. (hereinafter Edwards), the eye gaze direction is estimated by first determining the head pose angle and then by locating the iris position relative to the eye region based on a precise geometric model of eyes. In U.S. Pat. No. 7,046,924 of Miller, et al. (hereinafter Miller), the eye gaze direction and its path are estimated to identify an area of importance in images. In U.S. Pat. No. 7,197,165 of Ryan (hereinafter Ryan), a three-dimensional head model is utilized to estimate the head pose and gaze. The present invention employs basic ideas for its visual target estimation similar to the mentioned inventions; first estimate the head pose, and locate the eye and iris positions. The position of the irises relative to the localized eyes provides the data to estimate the gaze direction. However, we adopt a series of machine learning-based approaches to accurately and robustly estimate the gaze under realistic imaging conditions; a two-dimensional facial pose estimation followed by a three-dimensional facial pose estimation, where both estimations utilize multiple learning machines. The facial features are also accurately localized based on the estimated global facial geometry, again using combinations of multiple learning machines, and each take part in localizing a specific facial feature. Each of these machine learning-based estimations of poses or locations utilizes a set of filters specifically designed to extract image features that are relevant to a given estimation problem. Finally, the eye gaze estimates are interpreted differently with varying head pose estimates, to estimate the gaze direction and gaze target. Unlike most of the prior inventions, which focus on close-range visual targets, the present invention aims to estimate gaze regardless of distance, using a series of robust methods for face detection, pose estimation, and eye gaze estimation. To deal with the problem of gaze target estimation from a distance, the position of the head (the starting point of the gaze) is robustly estimated.
In summary, the present invention provides fully automatic face localization and facial feature localization approaches, for accurately extracting facial and transient features to estimate facial muscle actions due to emotional changes. It is a key novel feature of the present invention to train a learning machine based on the extracted emotion-sensitive features to estimate the facial muscle action; the emotion-sensitive features are designed to extract image features that are highly correlated with the facial expressions. The present invention shares the goal of estimating human response in relation to a given visual element similar to other rating approaches, but it adopts a unique method to analyze the final response based on the continuous emotion trajectory in the context of the given dynamic media programming.
The present invention also provides robust facial pose estimation and eye gaze estimation approaches to determine at which screen position a person is looking; it adopts a series of machine learning-based approaches to accurately and robustly estimate the gaze target under realistic imaging conditions, without using specialized imaging devices and without requiring close-range images or prior three-dimensional face models. The eye gaze is processed in the context of varying facial pose, so that the appearance changes of the eyes due to pose changes can be properly handled.
It is one of the novel features of the present invention to collect the emotional response of a person to a specific media content that the person is looking at, based on the association between the measured emotional response and the estimated gaze target. The method makes it possible for the temporal changes of emotion to be analyzed in the context of spatiotemporal media content.